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A  SURVEY  OF  TECHNIQUES  FOR  IMPROVING  COST 
ESTIMATES  OF  FUTURE  WEAPON  SYSTEMS 

SUMMARY 

The  incongruity  between  estimated  and  actual  costs  of 
today's  weapon  systems  indicates  a  need  for  cost  estimates 
which  more  accurately  predict  the  cost  of  future  weapon 
systems.  Estimates  made  near  the  beginning  of  a  development 
program  are  particularly  unreliable.  For  example,  the  cost 
of  developing  11  existing  weapon  systems  was  as  much  as 
seven  times  the  amount  originally  estimated.  A  study  of 
the  development  and  production  costs  of  33  weapon  systems 
showed  that  the  original  cost  estimates  were  180  to  220  per¬ 
cent  too  lew,  on  the  average,  even  after  price- level  and 
cost-quantity  adjustments  were  made.  The  problem  becomes 
acute  when  cost  estimates  are  requested  for  Air  Force  weapon 
systems  expected  to  become  operational  as  much  as  10  years 

in  the  future. 

,  •  ,■  ■■  .  ;  .  .■ 

Causes  for  cost  overruns  include  technical  complexity, 
technical  obsolescence,  schedule  slippages,  and  contractor 
optimism.  Frequently,  a  cost  estimate  is  inadequate  because 
it  does  not  include  all  the  necessary  elements  of  total 
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system  cost.  However,  by  far  the  greatest  cause  of  cost 
overruns  is  iequirements  uncertainty.  Sudden  and  unpredict¬ 
able  changes  in  technology,  enemy  plans,  and  U.S.  defense 
policies  can  change  the  requirement  for  a  -veapon  system 
drastically,  or  even  result  in  the  program's  cancellation. 

A  change  in  operational  concept  could  significantly  affect 
installations  and  personnel  costs.  The  other  majpdr  cause 
of  cost  overruns  is  technological  uncertainty.  The  size  of 
cost  overruns  is  directly  related  to  the  degree  of  tech¬ 
nological  advance  sought.  For  example,  missile  programs 
requiring  ambitious  advances  in  state-of-the-art  have  much 
larger  cost  overruns  than  do  cargo  and  tanker  aircraft 
programs  which  require  relatively  modest  innovations. 

An  adjustment  factor  can  be  applied  to  cost  estimates 
in  an  attempt  to  account  for  technological  and  requirements 
uncertainties.  The  adjustment  factor  is  the  ratio  between 
actual  and  estimated  costs  of  completed  weapon  systems 
requiring  advances  in  technology  which  are  comparable  with 
those  of  the  new  system.  Adjustment  factors  must  be  applied 
to  all  elements  of  the  total  system  cost,  a  long  and  tedious 
process.  Adjustment  factors  might  account  for  technological 
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uncertainty,  but  technical  breakthroughs,  enemy  capabilities, 
and  new  operational  concepts  cannot  be  accounted  for  by 
applying  a  factor  to  a  single-point  estimate. 

Subjective  probability  distributions  can  express  tech¬ 
nological  uncertainty,  but  not  requirements  uncertainty . 

The  number  of  routine  calculations  required  by  this  technique 
makes  extensive  computer  facilities  mandatory. 

The  cost-estimating  relation,  a  useful  tool  in  cost 
estimation,  is  a  statement  of  how  one  or  more  variables 
affect  another.  Even  limited  data  can  be  used  as  empirical 
evidence  in  the  derivation  of  cost-estimating  relations. 
However,  such  data  are  often  recorded  by  various  organiza¬ 
tions  under  many  different  and  vaguely  defined  cost  elements 
and  must  be  rearranged  so  that  like  costs  can  be  identified. 
The  lack  of  data  on  existing  missile  and  satellite  systems 
reduces  the  effectiveness  of  using  the  cost-estimating 
relation  in  determining  costs  of  future  Air  Force  weapon 
systems . 

Cost  sensitivity  analysis  is  best  described  as  a 
systematic  examination  of  the  changes  in  total  system  cost 
as  important  system  configuration  characteristics  are  varied 
over  their  relevant  ranges.  This  technique  best  provides 
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for  requirements  and  technological  uncertainties.  In  fact, 
it  can  even  provide  a  range  of  cost  estimates  for  a  weapon 
system  whose  ultimate  configuration  is  uncertain.  Cost 
sensitivity  analysis  requires  much  more  effort  than  a 
single-point  estimate,  but  long  or  complicated  cost¬ 
estimating  procedures  are  unnecessary.  This  technique  can 
be  used  readily  by  a  small  group  of  analysts  without  exten¬ 
sive  computer  facilities. 

Other  techniques,  singly  or  in  combination,  may  be 
useful  in  special  situations.  However,  cost  sensitivity 
analysis  is  the  single  most  effective  tool  for  estimating 
the  costs  of  future  weapon  systems. 
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A  SURVEY  OF  TECHNIQUES  FOR  IMPROVING  COST 
ESTIMATES  OF  FUTURE  WEAPON  SYSTEMS 


I .  INTRODUCTION 


Estimating  the  cost  of  future  weapon  systems  is  s.  diffi¬ 
cult  and  uncertain  process,  but  it  is  essential  for  budget 
preparation  and  for  major  program  decisions.  In  choosing 
between  possible  military  systems,  costs  of  the  alternatives 
must  bo  estimated  so  that  a  logical  selection  may  be  made. 
The  comparison  of  future  weapon  systems  requires  that  data 
be  gathered  on  the  relative  costs  and  effectivenesses  of  the 
systems. 

A  weapon  system  is  defined  by  Air  Force  Regulation  80-1 

(Reference  1)  as  follows: 

"Weapon  System.  Composed  of  equipment,  skills, 
and  techniques  the  composite  of  which  forms  an 
instrument  of  combat,  usually,  but  not  necessar¬ 
ily,  having  an  aerospace  vehicle  as  its  major 
operational  element.  The  complete  weapon  system 
includes  all  related  facilities,  equipment, 
material,  services,  and  personnel  required  for 
the  operation  of  the  system,  so  that  the  instru¬ 
ment  of  combat  can  be  considered  as  a  self- 
sufficient  unit  of  striking  power  in  its  intended 
operational  environment." 

Improved  cost  estimates  of  weapon  systems  are  becoming 
increasingly  important  because  many  of  the  nuw  systems  are 
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extremely  costly  and  require  a  substantial  portion  of  the 
Nation's  resources.  In  addition,  increasing  numbers  of 
systems  are  proposed  for  the  same  mission  or  objective. 

Weapon- system  cost  analysis  is  much  more  than  an  esti¬ 
mate  of  the  cost  of  the  weapon  itself.  Weapon  procurement 
costs  u:ay  be  relatively  small  compared  to  other  necessary 
expenditures  for  base  facilities,  training  of  personnel, 
and  operating  expenses.  Moreover,  in  the  comparison  and 
programing  of  systems  for  5  to  10  years  in  the  future,  the 
costs  of  research  and  development  must  be  taken  into  account. 
Recent  experience  indicates  these  costs  are  increasing  in 
relation  to  other  costs  and  may  be  expected  to  increase 
further  as  technological  change  accelerates.  Therefore,  the 
cost  of  the  complete  system  must  be  estimated  before  alterna¬ 
tives  can  be  compared.  All  related  support  costs  for  the 
entire  life  of  the  system  must  be  included  in  the  estimate. 
System  life  extends  from  the  beginning  of  development,  to 
system  activation,  and  on  through  operation  until  the  system 
leaves  the  active  inventory. 

This  report  reviews  the  inaccuracy  of  past  estimates  and 
evaluates  methods  for  improving  the  cost  estimates  for 
weapon  systems  expected  to  become  operational  as  much  as 
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10  years  in  rhe  future.  The  methods  are  studied  to 
determine  whether  theys 

-1  Provide  reliable  and  complete  estimates 
-2  Constitute  a  means  for  rapid  response  to  Air  Force 
requests 

-3  Can  be  used  by  small  groups  of  cost  analysts  without 
benefit  of  extensive  computer  facilities. 


II.  DISCUSSION 


A.  Cost-Estimate  Reliability 

Most  weapon- system  cost  estimates  made  before  procure¬ 
ment  usually  fall  short  of  the  actual  amounts  finally 
incurred.  Sometimes  the  actual  costs  are  many  times  the 
original  estimates.  Estimates  made  near  the  beginning  of  a 
development  program  are  particularly  unreliable. 

Table  1  presents  the  rati a  between  actual  and  estimated 
development  costs  for  11  weapon  programs  (Reference  2:  p.  22). 
Development  costs  include  all  the  expenditures  necessary  to 
bring  a  weapon  system  to  the  point  where  it  is  ready  for 
introduction  into  the  operational  force.  These  costs  include 
amounts  for  research,  test  vehicles,  ground- support  equip¬ 
ment,  and  for  all  the  activities  associated  with  the  test 
and  evaluation  of  the  weapon  system.  In  the  table,  the 
average  development  cost  factor  is  3.2.*  Examination  shows 
that  development  costs  in  the  programs  studied  were  as  much 
as  seven  times  the  original  estimates.  In  only  one  program 
were  actual  costs  less. 


*A11  averages  in  this  study  are  unweighted. 
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TABLE  1 


DEVELOPMENT  COST  FACTOR  IN  ELEVEN  WEAPON  PROGRAMS 


AVERAGE  .  3.2 


•Actual  cost  divided  by  original  cost  estimate. 


Table  2  summarises  some  data  on  estimates  of  production 
costs  (not  including  *he  cost  of  development)  for  22  major 
Air  Force  weapon  systems  (Reference  3*  pp.  467*469). 

Included  in  these  22  systems  are  fighters  (F-84,  F-89,  F-100, 
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F-102,  and  F-106) ,  bombers  (B-47,  B-52,  B-58) ,  cargo  and 
tanker  aircraft  (KC-135,  C-123A),  and  missiles  (BOMARC, 

SNARK,  THOR,  ATLAS,  and  TITAN)  .  The  ratio  of  the  latest 
available  estimate  or  actual  cost  to  the  original  estimate 
is  listed  for  each  weapon  system  (e.g.,  if  the  latest  cost 
estimate  were  twice  the  earliest  available  estimate,  the 
factor  would  be  2.0).  The  average  cumulative  cost  of  pro¬ 
duction  is  the  average  cost  of  procurement  of  a  given  number 
of  primary  flight  vehicles,  and  does  not  include  the  costs 
of  research  and  development,  other  initial  investments,  and 
operation  and  maintenance.  In  Table  2,  the  average 
unadjusted  cost  factor  is  6.5.  The  ratio  betweei,  the  latest 
estimate  and  the  earliest  estimate  of  the  cost  of  production 
for  these  vehicles  ranges  from  a  minimum  of  1  to  a  maximum 
of  57.6.  If  the  extreme  missile  case,  Number  1,  is  excluded, 
the  average  factor  for  the  21  remaining  items  is  4.1  instead 
of  6.5.  There  are  substantial  differences  in  the  averages 
of  the  four  classes  of  equipment.  The  average  factor  is  1.3 
for  cargo  and  tanker  aircraft;  for  missiles  it  is  17.1. 

The  raw  factors  presented  in  Table  2  are  unadjusted. 
Adjustments  are  made  for  changes  in  price  levels  or  for 
differences  between  the  actual  number  of  vehicles  procured 
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and  the  number  of  vehicles  that  were  to  be  procured  at  the 
time  the  early  cost  estimates  were  made.  Changes  in  price 
levels  have  generally  been  upward?  therefore,  the  actual 
costs  should  be  deflated.  The  cumulative  average  cost  of 
production  is  a  decreasing  function  of  total  output.  Since 
actual  total  output  usually  is  very  different  from  the  out¬ 
put  which  was  originally  anticipated,  the  original  cost 
estimate,  based  on  a  different  procurement  level,  is 
adjusted  to  reflect  the  actual  number  of  vehicles  procured. 

As  shown  by  Table  2,  the  effect  of  the  adjustments  is 
a  reduction  in  the  size  of  the  factors.  The  average  factor 
for  all  22  items  is  reduced  from  6.5  to  2.8.  Most  of  the 
reduction  is  a  result  of  the  output  adjustment  rather  than 
of  the  price-level  adjustment.  However,  even  after  the 
adjustments  were  made  on  the  raw  factors  in  Table  l,  the 
actual  costs,  on  the  average,  exceeded  the  original  estimates 
by  180  percent.  Individual  production  costs  were  as  much  as 
10.5  times  the  original  cost  estimates. 

Tables  1  and  2  show  that  original  cost  estimates  were 
180  to  220  percent  too  low  on  the  average,  even  after  price- 
level  and  cost-quantity  adjustments  were  made.  For 
individual  systems,  actual  development  and  production  costs 
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were  as  much  as  6  and  10  times  greater  than  the  early  cost 
estimates.  When  cost  estimates  are  in  error  by  such  large 
margins,  the  decision  maker  has  difficulty  in  selecting 
logically  between  alternative  weapon  systems. 

B.  Analvsis_of  Factors  Influencing  Cost  Estimates 

1.  Technical  Complexity 

One  of  the  outstanding  features  of  today's  weapon-system 
programs  is  technical  complexity.  This  technical  complexity 
includes: 

-1  A  large  number  of  technical  problems  for  each  new 
weapon  system 

-2  The  interrelation  among  the  technical  problems 
-3  The  large  number  of  components  in  the  system  in 
light  of  reliability  requirements. 

All  of  these  obstacles  may  be  encountered  within  any  one 
program . 

2.  lfi.ghnicflL_9b80lggC8ngfi 

Technical  obsolescence  is  another  problem  of  today's 
technical  environment.  Weapon- system  programs  must  often 
be  changed  or  canceled  as  the  technical  state-of-the-art 
advances.  The  SNARK  system  is  an  example.  The  SNARK  pro¬ 
gram,  begun  in  1946,  was  for  the  development  of  an 
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and  have  a  range  of  6,000  milea.  By  1959,  the  less- 
vulnerable  ballistic  missiles  had  made  SHARK  obsolete.  The 
program  was  terminated  after  an  expenditure  of  about 
$700  million  and  procurement  of  only  one  operational  squadron. 
Similarly,  the  NAVAHO,  an  Air  Force  supersonic  air-breathing 
missile  costing  about  $700  million,  was  made  obsolete  before 
it  entered  production  by  the  development  of  the  ballistic 
missile. 

3.  schedule.  SJULeeagga 

Another  possible  cause  of  cost  overruns  in  development 
programs  is  long  development  time.  The  data  reviewed 
indicate  that  cost  overruns  decreased  as  greater  emphasis 
was  placed  on  minimizing  development  time.  As  can  be  seen 
in  Table  3  and  Figure  1,  the  greater  the  schedule  slippage 
in  a  program,  the  larger  the  cost  overrun  (Reference  2: 
p.  442). 

4»  Timing  of  Cost  Estimate 

Another  factor  that  should  be  considered  is  the  timing 
of  the  cost  estimate.  Was  the  cost  estimate  made  before,  or 
some  time  during,  the  development  program?  As  might  be 
expected,  data  indicate  that  average  cost  overruns  tend  to 


TABLE  3 


CORRELATION  OP  COST  OVERRUN  FACTORS  WITH  TIME  SLIPPAGE  FAILURES 
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decrease  if  cost  estimates  are  made  lahe  in  the  development 
program.  This  study  only  considers  cost  estimates  made 
early  in  the  development:  program,  because  its  purpose  is  to 
evaluate  methods  for  estimating  the  cost  of  potential  Air 
Force  systems  that  could  become  operational  10  years  in  the 
future. 

5 .  Competitive  Optimism 

When  a  contractor  learns  of  Air  Force  interest  in  a  new 
concept*  he  prepares  proposals  which  include  cost  estimates. 

The  contractor  usually  reflects  his  desire  to  win  the  con- 

\ 

tract  by  outbidding  the  competition  with  optimistic  cost 
estimates  for  initial  development.  However,  not  all  of 
this  optimism  is  due  to  the  competitive  strategy  of  the 
contractor.  Sometimes  the  Mr  Fcrce  and  its  buying  agencies 
discourage  realistic  cost  estimates  because  the  budget  may 
be  inadequate  to  support  the  new  program.  In  other  cases. 

Air  Force  requirements  have  placed  a  premium  on  achieving 

‘  -  •  \  *  * 

the  maximum  possible  advance  in  state-of-the-art,  thereby 
encouraging  the  contractor  to  prepare  optimistic  technlc.il 

proposals.  It  seems  impossible  to  separate  the  cost  over- 

'  *,»• 

run  caused  by  competitive  optimism  from  that  caused  by 
technical  uncertainty.  Technical  progress  occurs  between 
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the  time  the  initial  contractor  estimates  are  made  and 
when  the  program  begins.  However,  there  is  little  doubt 
that  as  the  intensity  of  competition  increases,  the  cost 
estimates  in  the  contractor  proposals  become  more 
optimistic. 

The  cost  analyst  is  confronted  with  the  task  of  making 
cost  estimates  of  advanced  systems  with  vary  little  infor¬ 
mation  other  than  the  optimistic  contractor  cost  estimates. 
He  must  somehow  adjust  these  contractor  estimates  to  take 
into  account  the  competitive  optimism  of  the  contractor,  ? f 
he  intends  to  use  the  contractor's  data.  The  better  altorna 
tive  would  be  for  the  cost  analyst  to  make  ar.  independent 
cost  estimate  based  upon  the  best  available  technical 
information  and  Air  Force  requirements  for  the  weapon  system 
6.  (Paragraph  6  intentionally  deleted  from  documents  for 
open  publication.) 
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7 .  Inaccurate  Cost  Estimates 


Another  source  of  error  in  cost  estimates  can  be  the 
incompleteness  of  the  cost  analysis.  Frequently,  the  cost 
analyst  will  produce  poor  cost  estimates  because  he  has 
failed  to  gather  all  the  ava.  lable  technical  and  operational 
data  on  the  proposed  system.  Many  times,  the  estimate  of 
total  system  costs  does  not  include  all  the  necessary  ele¬ 
ments  of  system  costs.  For  example,  the  cost  of  a  missile 
system  night  include  the  costs  of  the  missile,  the  launcher, 
and  the  silo,  but  fail  to  include  a  prorated  share  of  the 
cost  of  the  launch  control  center.  It  is  important  that 
weapon- system  costs  be  comprehensive,  so  that  sound 
decisions  can  be  made.  It  is  equally  important  that  all  the 
assumptions  cn  which  the  cost  estimates  are  based  be  pre¬ 
sented  to  the  decision  maker.  Intrinsic  errors  may  also  be 
expected  in  the  basic  data  used  in  cost  analysis  and  ir. 
cost-estimating  relations. 

8*  Price  and_Procurement  Levels 

Changes  in  price  levels  during  the  development  of  a 
weapon  system  are  relatively  small  sources  of  error,  and 
cost  estimates  are  usually  presented  in  current  dollar 
values.  The  crudeness  of  present  cost  estimates  makes  cf 
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questionable  value  the  added  refinement  of  predicting  price 
levels  durinc  the  life  of  the  weapon  system.  As  a  matter 
of  general  interest,  it  may  be  desirable  to  include  the 
incremental  costs  caused  by  assuming  several  levels  of 
inflation  during  the  life  of  the  weapon  system.  However, 
even  if  price-level  changes  could  be  forecast,  the  effect 
on  the  relative  costs  of  alternative  systems  would  generally 
be  much  less  than  that  of  other  uncertainties.  Such  eco¬ 
nomic  computations  should  be  supplemental  to  the  basic  cost 
analysis. 

The  actual  costs  of  any  weapon  system  depend  very  much 
or.  the  quantities  procured.  Large  cost  overruns  are  caused 
by  cost  estimates  based  upon  one  particular  predicted  force 
size  or  level  of  procurement.  If  for  any  reason  a  different 
number  of  items  is  procured,  large  errors  in  cost  estimates 
result.  A  simple  solution  to  this  problem  is  the  presenta¬ 
tion  of  cost  estimates  for  a  wide  range  of  procurement 
levels,  thereby  giving  the  decision  maker  the  means  for 

,3  ^  • 

quickly  and  accurately  determining  the  differences  in  total 
system  costs  as  he  changes  the  size  of  the  force.  Unit 
ccsts  generally  decrease  as  the  quantity  produced  increases, 
and  this  functional  relation  is  a  necessary  part  of  any  cost 
analysis. 
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9.  Cost-Estimating  Uncertainty 

There  are  several  possible  sources  of  cost-estimating 
uncertainty.  Cost-estimating  relations  used  in  cost 
analysis  cannot  be  assumed  to  hold  exactly.  In  estimating 
a  certain  cost  component  as  a  function  of  many  variables, 
it  cannot  be  assumed  that  the  variables  will  predict  the 
particular  cost  with  certainty.  The  observations  used  in 
deriving  cost-estimating  relations  invariably  contain 
errors.  In  costing  advanced  Air  Force  systems,  cost¬ 
estimating  relations  from  past  experience  are  sometimes 
used.  Extrapolations  beyond  the  range  of  the  sample  or 
data  base  from  which  the  estimating  relation  was  derived 
are  another  source  of  uncertainty.  Cost-estimating  errors 
may  arise  because  of  the  use  of  techniques  which  involve  a 
considerable  amount  of  aggregation.  An  important  cost  ele¬ 
ment  may  be  overlooked  if  the  aggregations  are  large.  Large 
aggregations  also  make  cost  comparisons  of  alternatives 
difficult. 

1°.  Air,  force  Requirements  Uncertainty 

Air  Force  requirements  for  a  future  weapon  system 
drastically  affect  a  development  program  and  its  associated 
costs.  Sudden  and  unpredictable  changes  in  technology, 
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enemy  plans,  and  U.S.  defense  policies  might  require  pro¬ 
gram  redirection  or  even  cancellation.  Redesign  of  the 
weapon  system  might  prove  necessary  and  could  produce  a 
different  configuration  of  the  weapon  system  and  all  of 
its  associated  support  equipment.  The  uncertainty  is 
further  compounded  by  the  fact  that  the  requirement  for  an 
advanced  weapon  is  based  upon  intelligence  estimates  of  the 
weapons  possessed  by  a  potential  enemy.  Intelligence  esti¬ 
mates  are  subject  to  a  considerable  margin  of  error  and 
change  suddenly  as  new  information  becon.es  available. 

Changes  in  the  configuration  of  the  weapon  system  are 
of  two  basic  types.  One  change  involves  hardware  charac¬ 
teristics.  For  example,  a  new  engine  may  be  included  on 
a  strategic  bomber.  The  other  involves  the  system's  opera¬ 
tional  concept.  For  example,  do  the  strategic  bombers  use 
existing  Strategic  Air  Command  bases?  or  are  they  dispersed 
to  remote  locations? 

There  are  many  possible  reasons  for  changes  in  a 
system's  configuration.  The  original  design  may  fail  to 
produce  the  required  performance  characteristics.  Required 
performance  characteristics  may  be  changed  with  a  resultant 
change  in  hardware  specifications.  An  attempt  may  be  male 


to  acquire  the  system  sooner  than  was  originally  intended 
by  substituting  resources  for  time.  The  strategic  situation 
may  change,  producing  changes  in  system  deployment.  For 
example,  a  higher  degree  of  dispersal  or  alert  capability 
may  be  required  to  reduce  vulnerability  of  the  system  to 
surprise  attack  in  a  new  strategic  situation.  Such  changes 
of  operational  concept  could  significantly  affect  installa¬ 
tions  and  personnel  costs  which  are  substantial  elements  of 
total  system  cost.  Of  course,  closely  related  to  these 
possible  reasons  for  changes  in  a  system's  configuration  are 
the  unforeseen  technical  difficulties  that  will  be 
encountered  in  the  development  programs  of  future  weapon 
systems. 

Technical  problems  of  many  kinds  can  be  expected  in  a 
typical  weapon-system  program,  and  each  of  these  unexpected 
difficulties  has  an  effect  on  the  contractor's  ability  to 
meet  original  time,  quality,  and  cost  predictions.  Table  2 
revealed  a  direct  correlation  between  the  average  cost 
factors  and  different  classes  of  equipment.  Ths  smallest 
average  coat  factor  was  computed  for  the  cargo  or.d  tanker 
class  and  the  largest  for  the  missile  c'ass.  The  performance 


requirements  for  new  cargo  and  tanker  aircraft  are  usually 
less  than  what  has  already  been  achieved  by  bomber  aircraft. 
The  engines  for  the  Cc.rgo  and  tanker  aircraft  are  usually 
"off-the-shelf"  itoms.  In  terms  of  both  performance  and 
physical  characteristics  of  the  equipment,  comparatively 
modest  innovations  are  required  in  cargo  and  tanker  develop¬ 
ments.  On  the  other  hand,  the  technology  which  characterized 
the  missile  programs  in  Table  2  called  for  ambitious 
advances.  Complex  guidance  and  control  systems  and  advanced 
propulsion  techniques  were  required,  and  performance 
requirements  were  an  order  of  magnitude  greater  than  had 
been  achieved  before. 

The  correlation  between  degree  of  technical  advance 
sought  and  cost  overrun  factors  is  demonstrated  in  Table  4 
(Reference  3:  p.  472).  Technical  experts  classified  the 
development  programs  in  the  table  as  small,  medium,  or 
large,  according  to  the  technical  advance  sought.  The 
average  cost  factor  shown  is  an  increasing  function  of  the 
size  of  the  technical  advance.  Programs  with  small  advances 
had  an  average  factor  of  1.4;  programs  with  medium  advances, 
1.8;  and  programs  with  large  advances,  4.2.  However,  there 
is  some  inconsistency  in  the  cost  factors  of  the  individual 
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weapon  systems.  Several  of  the  programs  with  small  and 
medium  technical  advances  had  cost  factors  greater  than 
several  of  the  programs  with  large  technical  advances. 

To  further  ascertain  the  extent  of  the  correlation 
between  technical  advance  and  cost  overrun  factors,  the 
development  programs  in  Table  1  were  assigned  state-of-the- 
art  indices  as  shown  in  Table  5.  The  state-of-the-art  index 
measures  the  technical  achievement  and  innovation  required 
to  accomplish  the  program's  objective.  The  values  of  the 
indices  range  from  0  to  100.  A  development  program  with  a 
0  index  indicates  that  no  technical  advance  is  required 
beyond  the  technology  of  systems  already  in  production.  An 
index  of  100  represents  a  development  prcgrara  which  would 
require  significant  and  unforeseen  breakthroughs  in  system 
technology.  Programs  with  indices  between  1  and  99  are 
within  the  state-of-the-art,  and  the  magnitude  of  the  index 
in  these  case3  varies  inversely  with  the  amount  of  knowledge 
and  experience  available  for  each  program. 

Figure  2  presents  data  on  the  11  development  programs, 
with  state-of-the-art  indices  plotted  as  the  independent 
variable  (Reference  2:  p.  437).  A  positive  correlation 
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TABLE  5 

DEVELOPMENT  COST  FACTOR  VERSUS  STATE-OF-THE-ART  INDEX 


Fr  oct  ram 

Development 

Cost  Factor* 

State-of-the-Art 

Index 

A 

4.0 

90 

B 

3.5 

65 

C 

5.0 

92 

D 

2.0 

55 

E 

7.0 

90 

F 

3.0 

80 

G 

2.0 

50 

H 

2.4 

85 

I 

2.5 

60 

J 

.7 

80 

K 

3.0 

60 

AVERAGE 

3.2 

•Actual  cost  divided  by  original  cost  estimate. 


was  obtained  between  state-of-the-art  exploitation  and 
development  cost  factors,  but  the  correlation  coefficient 
is  not  high. 

From  the  data  shown  in  Tables  4  and  J  <ar»d  in  Figure  2, 
it  seems  reasonable  to  conclude  that  technological  uncer¬ 
tainty  is  a  ma3or  cause  of  production  and  development  cost 
overruns . 

12 .  Corroborating  Studios-. 

Study  and  analysis  of  historical  cort  data  have  indicated 
that  requirements  and  technological  uncertainties  are  the 
primary  causes  of  errors  in  the  cost  estira.  »  of  future 
weapon  systems.  These  uncertainties  are  so  closely  related 
that  it  would  be  unreasonable  to  consider  them  separately. 
Cost-estimating  uncertainty  caused  by  intrinsic  errors  in 
costing  a  fixed  configuration  is  small  in  ccanpc.rison  with 
requirements  and  technologies]  uncertainties.  The  conclu¬ 
sions  of  other  studies  support  this  statement. 

One  of  the  first  works  in  this  area  was  prepared  by 
Eugene  Brussell  at  The  RAND  Corporation  (quoted  in 
Reference  4;  p.  7} .  He  concluded  that  the  pri.ir.ary  cause 
of  cost  overruns  is  that  the  equipment  .being  costed  changed 
after  the  estimate*  were  completed  and  that  most  of  these 
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changes  resulted  in  higher  costs.  However,  Brussell  was 
primarily  concerned  with  systems  hardware  costs  rather  than 
cotal  cost.  Marshall  and  Heckling,  after  studying  Brussell' s 
work  and  Robert  Summers'  expansion  of  it,  reached  similar 
conclusions  regarding  the  main  source  of  error  in  cost 
analysis.  fhey  pointed  out  that  early  cost  estimates  for 
producing  or  developing  something  new  are  usually  based  on 
the  initial  design  and  program  plans.  As  development  pro¬ 
ceeds,  the  initial  designs  and  plans  are  almost  invariably 
changed,  either  because  of  unforeseen  .  ^al  difficulties 

or  because  the  customer  decides  it  is  essential  that  the 
equipment  be  modified  to  keep  pace  with  changing  predictions 
of  enemy  capabilities,  new  operational  concepts,  and  new 
technological  possibilities. 

In  theory,  it  would  be  possible  to  divide  into  two  parts 
the  total  error  in  cos*  estimates  as  they  are  prepared: 

(1)  errors  in  the  cost  of  the  configuration  supplied  by  the 
cost  estimator  (i.e.,  the  intrinsic  error  in  cost¬ 
estimating)  ,  and  (2)  changes  in  the  configuration  as  devel¬ 
opment  progresses.  In  practice,  it  has  not  been  possible 
to  carry  out  this  separation.  However,  in  the  costing  of 
most  major  items  of  military  equipment,  the  intrinsic  errors 
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tend  to  be  small  in  relation  to  errors  caused  by  configura¬ 
tion  changes  (Reference  4:  p.  8).  Scherer  and  Peck 
concluded  that  one  of  the*  primary  causes  of  coat  overruns  is 
the  unexpected  difficulty  :aused  by  technical  uncertainties 
(Reference  2) .  Hitch  and  McKean  agreed  that  technological 
uncertainty  is  the  major  source  of  error  in  cot;t  estimates 
of  future  weapon  systems  (Reference  5s  p.  189). 

Another  study  considered  variations  in  total  system 
cost  stemming  from  changes  in  system  operational  concept 
(Reference  4:  pp.  8-9) .  The  study  pertained  exclusively 
to  intercontinental  ballistic-missile  systems.  It  provides 
data  on  more  recent  systems  than  does  the  Brussell  study. 

The  study  concluded,  as  did  Brussell,  that  the  main  source 
of  variation  in  cost  estimates  is  requirements  uncertainty. 
It  is  apparent  from  the  study  that  in  1954  to  1960,  fluc¬ 
tuations  in  cost  estimates  were  occasioned  most  frequently 
by  changes  in  operational  and  organizational  concept.  In 
addition  to  examining  variations  in  total  system  cost  for 
ICBM  systems,  the  study  considered  various  components  of 
the  total.  For  example,  estimates  of  personnel  and  training 
costs  were  shown  to  be  clearly  dependent  upon  varying  con¬ 
cepts  of  ccenbat  employment  for  the  ICBM  force.  Considerable 
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fluctuations  in  5-year  personnel  and  training  cost  estimates 
pv»r  operational  missile  have  occurred  as  manning  concepts 
have  responded  to  changes  in  the  salvo  capabilities  required. 

Like  the  work  of  Brussell,  this  study  presented  cost 
variations  in  quantitative  terms,  but  the  explanations  of 
the  variations  were  mostly  qualitative.  Even  though  essen¬ 
tially  qualitative,  these  explanations  offer  proof  that  most 
of  the  variations  in  system  cost  may  be  attributed  to 
requirements  uncertainty  rather  than  cost-estimating 
uncertainty. 

The  question  still  remains  as  to  the  relative  magnitudes 
of  the  two  types  of  uncertainty.  Attempts  have  been  made  to 
answer  this  question  by  examining  caae  histories  of  past 
weapon  systems,  but  the  attempts  were  unsuccessful.  The 
data  sources  available  do  not  permit  a  quantitative 
identification  of  these  two  sources  of  error  in  cost  esti¬ 
mates  of  weapon  systems. 

In  another  study,  Armen  Alchian  examined  several  cases 
where  requirements  uncertainty  was  not  a  major  factor 
(Reference  9:  pp.  10-11) .  The  study  focused  on  items  and 
methods  of  production  that  were  well  within  *the  state-of- 
the-art.  He  was  essentially  examining  cost-estimating 
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uncertai.ity .  Although  the  datb  are  neither  recent  nor  very 
broad  in  scope,  they  do  provide  some  basis  for  a  quantita¬ 
tive  statement  on  cost-eBtinating  uncertainty.  There  is  no 
reason  to  believe  that  the  age  or  the  amount  of  data 
preclude  their  validity.  On  the  basis  of  these  data,  it  is 
concluded  that  the  variation  in  cost  estimates  attributable 
purely  to  cost-estimating  uncertainty  might  average  20  to 
30  percent.  On  the  other  hand,  the  worKs  of  Brussell, 
Summers,  and  others  indicate  that  variation  in  cost  estimates 
can  average  200  percent  or  higher,  while  for  individual  sys¬ 
tems,  actual  development  and  production  costs  were  as  much 
as  *  and  11  times  the  original  cost  estimates.  If  these 
data  are  accepted  as  reasonable,  the  conclusion  is  that 
variation  in  cost  estimates  attributable  to  cost-estimating 
uncertainty  is  small  relative  to  that  associated  with 
requirements  uncertainty. 

C •  Evaluation  of  Post-Estimating  Techniques 
1 .  Objective 

There  is  a  need  for  a  cost-estimating  technique  that 
will  take  into  account  technological  and  requirements  uncer¬ 
tainties.  The  technique  must  be  usable  by  a  small  group 
of  cost  analysts  with  technical  backgrounds  and  very  limited 
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data-processing  equipment.  It  must,  require  a  minimum  of 
calculations  so  that  reliable  cost  estimates  on  future 
weapon  systems  and  concepts  can  be  provided  to  the  Air  Force 
on  short  notice. 

2 .  Description  of  Available  Techniques 
a*  The  Adjustment  Factor 
U)  &£S£ijjofcigri 

One  possible  solution  to  the  problem  of  ret*,  rements 
and  technological  uncertainties  in  cost  estimates  is  the 
use  of  adjustment  factors.  The  adjustment  factor  is  a 
cost  overrun  factor,  defined  earlier  (Section  Il-A)  as 
the  ratio  between  the  actual  cost  of  the  system  and  the 
estimated  cost.  The  adjustment  factor  is  applied  to 
the  cost  estimate  in  an  attempt  to  account  for  uncer¬ 
tainties  in  requirements  and  technology.  The  factor  to 
be  applied  to  the  cost  estimate  would  depend  upon  the 
kind  of  system  being  developed  and  the  degree  of 
technological  advance  associated  with  the  system.  On 
the  average,  the  cost  overrun  factor  increases  as  the 
technological  advance  increases.  (See  Figure  2,  p.  26.) 
It  is  suggested  that  adjustment  factors  similar  to  the 
average  cost  overrun  factors  could  be  applied  to  cost 
estimates . 
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Engene  Brussell  is  responsible  for  gathering  the 
historical  data  on  past  weapon-system  programs  as  shown 
in  Table  2  (p.  7)  and  in  Table  4  (repeated  on  p ,  34  for 
convenience).  In  his  analysis,  he  computed  increases 
in  major-equipnent  cost  estimates  by  comparing  estimates 
made  early  in  a  weapon- system  program  with  estimates 
made  late  in  the  program.  He  concluded  that  the  main 
benefits  of  his  study  were  a  clearer  understanding  of 
which  systems  are  most  likely  to  experience  large  cost 
biases ,  and  some  insight  about  the  structure  of  the 
problem  of  uncertainty  in  cost  analysis  (Reference  4: 
p.  7)  .  He  indicated  that  it  is  not  possible  to  develop 
adjustment  factors  which  could  be  applied  mechanically 
ai.d  which  would  be  valid  under  a  wide  range  of 
circumstances. 

Bruscdl’s  study  had  some  definite  limitations. 

The  analysis  considered  only  major-equipment  costs,  not 
total  system  costs.  The  study  was  made  when  cost- 
estimating  techniques,  methods,  and  concepts  were  in 
their  infancy.  There  is  no  way  of  determining  who  made 
the  cost  estimates  or  for  what  purpose. 
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TABLE  4 


PRODUCTION  COST  FACTORS,  CLASSIFIED  ACCORDING 
TO  DEGREE  OF  TECHNOLOGICAL  ADVANCE 


Small  Advance 

jJBHHPHSgjjjjj 

r - - - 

Laroo  Advance 

Weapon 

Type* 

Factor 

Weapon 

Type* 

Factor 

Weapon 

Type* 

Factor 

C  .... 

..  1.5 

B  *  •  •  • 

..  2.8 

B  •  »  •  • 

.  1.2 

F  .... 

CO 

• 

H 

• 

• 

F  .... 

..  2.5 

F  •  •  •  • 

.  1.0 

c 

Vi  •  •  •  • 

..  .9 

F  •  •  •  • 

..  2.0 

F  •  •  •  • 

.  .9 

c  .... 

..  1.5 

F  .... 

..  1.2 

B  .... 

.  5.1 

c  .... 

.  .  .9 

F  .... 

. .  .8 

.  1.0 

F  .... 

..  1.5 

M  .... 

..  1.3 

.10.5 

... 

1H 

.  3.9 

M  •  •  •  • 

.  3.6 

M  .... 

.  7.1 

M  .... 

.  .  7.1 

AVERAGE 

1.8 

4.2 

^  ■■■■■  «  '  —  1  ■  . . . 

*b  »  bomber,  C  »  cargo  aircraft  or  tanker,  F  =  fighter, 

and  M  »  missile. 
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Care  must  be  taken  in  interpreting  the  average 
factor  increases.  For  example,  in  Table  4  the  average 
production  cost  factor  for  a  "medium*  technological 
advance  is  1.3.  This  is  a  simple  average,  and  does  not 
reflect  the  relative  importance  or  type  of  weapon  system. 
Four  of  the  six  entries  in  this  column  are  fighters, 
one  is  a  bomber,  and  one  a  missile.  It  is  doubtful 
that  this  adjustment  factor  could  be  applied  to  a 
bomber  or  ricsile  system  with  confidence. 

Based  upon  further  analysis  of  the  data  assembled 
by  brussell,  Robert  Summers  developed  a  "magic  formula" 
for  deriving  an  adjustment  factor  for  major-equipment 
cost  estimates.  Summers  claimed  that  although  the 
formula  may  not  work  very  well  in  any  one  particular 
case,  its  repeated  use  for  a  large  number  of  cases 
will  result  in  equipment  cost  estimates  which,  on  the 


average,  will  be  more  accurate  than  those  obtained 


without  the  use  of  the  formula 


The  important  parameters  which  are  included  in  the 


'magic  formula"  are: 


-1  The  time  the  estimate  is  made  m  relation  to  the 


development  program 
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-2  The  decree  of  technological  advance  required 
-3  The  length  of  the  aevelopment  period. 

When  values  of  the  above  variables  for  a  particular 
advanced  system  are  substituted  into  the  formula,  the 
result  is  an  adjustment  factor.  If,  for  example,  the 
formula  produces  a  value  of  2.5  for  a  particular  system 
whose  major-equipment  cost  is  estimated  to  be  X,  the 
adjusted  estimate  would  be  2.5X.  The  formula,  althovqh 
complex,  simply  states  that  the  adjustment  factor  for 
systems  requiring  only  minor  technological  advances 
and  short  development  time  will  be  close  to  unity. 

But,  when  major  technological  advances  are  sought  and 
development  time  is  long,  the  adjustment  factor  will 
be  greater  than  unity,  probably  between  2  and  3. 

Certainly,  it  is  not  possible  to  correct  a  cost 
estimate  perfectly  by  multiplying  it  by  a  factor  which 
reflects  the  error  of  similar  estimates  in  tne  past. 
However,  if  the  estimates  are  not  corrected,  the  data 
in  Table  4  indicate  that  for  a  program  requiring  major 
technological  advances,  the  cost  estimate  will  be  too 
low  by  an  average  factor  of  4.2.  Multiplication  by  a 
debiasing  factor  of  3  or  4  will  not  make  ♦'he  estimate 


correct,  but  the  revised  estimate  is  likely  to  be 
closer  to  the  actual  costs  than  the  original,  unadjusted 
estimate . 

Sumners'  analysis  is  a  refinement  of  Brusuell's 
earlier  work  but  is  nonetheless  subject  to  most  of  the 
same  limitations.  The  data  base  is  restricted  with 
respect  to  both  quantity  and  quality.  The  analysis 
is  confined  to  major- equipment  costs.  The  formula  is 
complex  and  not  easy  to  apply.  Another  problem  arises 
from  the  probable  difficulty  of  arriving  at  reasonable 
estimates  for  the  degree  of  technological  uncertainty  to 
be  substituted  in  the  formula.  Major- equipment  cost  is, 
in  many  cases,  only  a  small  part  of  the  total  system 
cost,  which  includes  research,  development,  test,  evalu¬ 
ation,  initial  investment,  and  operation  of  not  only 
the  primary  vehicles,  but  also  all  the  associated 
support  equipment,  facilities,  and  personnel.  It  is 
necessary  to  describe  all  of  the  cost  elements  of  a 
weapon  system  so  that  the  decision  mcker,  faced  with 
selecting  a  system  from  among  several  possibilities, 
will  have  the  total  cost,  not  just  the  procurement  cost 
of  the  aircraft,  satellite,  or  missile. 
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In  order  to  prepare  estimates  of  total  system  cost, 
the  analyst  should  have  a  comprehensive  list  of  the 
categories  to  be  included.  Table  6,  an  example  of  such 
a  list,  divides  total  cost  into  three  ma^or  cateogires: 
research,  development,  test,  and  evaluation;  initial 
investment;  and  annual  operation.  Subcategories  are 
provided  which  outline  total  system  cost  in  further 
detail.  The  list  is  intended  to  be  as  inclusive  as 
possible  to  preclude  the  omission  of  significant  ele¬ 
ments  of  thr  total  cost.  The  cost  categories  should  be 
structured  such  a  way  that  those  elements  of  the 
system  which  nave  the  greatest  impact  on  total  cost  can 
be  easily  determined. 

There  are  advantages  to  a  detailed  listing  of  cost 
elements.  Fi *st.  the  costs  of  individual  elements  of 
two  competing  weapon  systems  may  lie  more  accurately 
compared  if  both  have  been  costed  with  the  same  format. 
Perhaps  only  major  cost  categories  are  available  for  a 
weapon  system  which  is  to  be  compared  to  another  system 
If  the  data  for  the.  other  system  have  been  divided  into 
smaller  elements,  it  might  be  possible  to  aggregate  or 
recombine  the  email  homogeneous  units  to  match  the 


TABLE  6 


WEAPON-SYSTEM  COST  CATEGORIES 


Research,  Deve lopment.  Test,  and  Evaluation 

Design  and  Development 

Test  and  Evaluation 

Management  and  Technical  Direction 


Initial  Investment 


Installations 
Primary  Equipment 
Support  Equipment 
Spares 

Initial  Training 
Miscellaneous 


Operation 

Equipment  Replacement  and  Maintenance 
Installations  Maintenance 
Pay  and  Allowances 
Training 

Fuels,  Lubricants,  and  Propellants 
Miscellaneous 
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major  cost  categories.  By  defining  categories  or  sub¬ 
categories  to  a  reasonable  degree,  the  cost  analyst  is 
less  likely  to  overlook  an  important  cost  element. 

If  adjustment  factors  were  applied  only  to  the 
major-equipment  category,  many  other  important  cost 
elements  of  the  total  system  cost  would  be  neglected. 

The  following  descriptions  of  the  subcategories  listed 
in  Table  6  are  intended  to  emphan-ze  tne  scope  of  total 
system  cost. 

Under  the  major  category  of  research,  development, 
test,  and  evaluation,  the  design  and  development 
include  preliminary  design,  applied  research,  mockups, 
test  equipment,  nonairborne  instrumentation,  additional 
plant  facilities,  and  captive  test  operations.  System 
test  encompasses  the  costs  of  flight  test  vehicles, 
tost  operations,  test  ground-support  equipment,  and  test 
facilities.  Systems  management  and  technical  direction 
are  usually  performed  by  a  corporation  and  include 
systems  engineering  and  technical  direction  to  contractors 
engaged  in  the  development  of  the  various  subsystems. 

Under  the  major  category  of  investment,  installa¬ 
tions  could  include  launch  pads,  runways,  new 
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maintenance  shops,  and  personnel  facilities.  The  primary- 
equipment  category  is  the  aircraft  or  missile  assigned 
to  combat  organizations.  Support  equipment  could  include 
launchers,  control  centers,  cables,  cranes,  and  trucks 
foi  a  missile  system.  An  allowance  for  spare  parts  and 
major-equipment  spares  is  also  assumed.  Initial  training 
costs  cover  the  formal  training  of  personnel  and  any 
line  firings  of  mis  .  ‘s  for  training  purposes.  The 
miscellaneous  investment  category  includes  items  such  as 
transportation  of  initial  supplies  and  equipment  and  the 
travel  expenses  of  personnel  to  the  operational  bases. 

Operating  costs  are  recurring  annual  expenditures 
for  operation  .nd  maintenar~e  of  the  system  onoe  it  has 
Leen  initiated  into  service.  Equipment  replacement  and 
maintenance  is  an  important  subcategory  of  operation 
cost.  Fo;.  mple,  significant  replacement  costs  would 
be  incurred  .  i  certain  number  of  satellites  in  orbit 
were  required  despite  high  attrition  rates. 

The  cost  of  pay  and  allowances  is  determined  by 
applying,  appropriate  factors  to  the  kinds  and  numbers 
of  personnel  required.  The  miscellaneous  category  in¬ 
cludes  transportation  of  replacement  equipment  and 
personnel. 
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If  adjustment  factors  are  to  be  used,  they  must  be 
develop'd  for  all  theso  elements  of  total  system  cost, 
not  just  the  major- equipment  category.  There  is  prac¬ 
tically  no  data  base  which  could  be  ueed  to  derive  the 
formulas.  Even  if  such  data  existed,  a  different 
rormula  would  be  required  for  each  cost  element, 
resulting  in  a  cumbersome  and  time-consuming  technique. 

It  would  be  extremely  difficult  to  formulate  cost 
elements  having  adequate  flexibility  for  application  to 
a  wide  variety  of  Air  Force  systems. 

(2)  Analysis 

Adjustment  factors  can  be  applied  to  cost  estimates 
to  take  into  accou.it  technological  uncertainties  of  a 
weapon  system,  but  technical  difficulties  are  only  cr.e 
part  of  the  uncertainty  problem.  Technical  breakthroughs, 
enemy  capabilities,  and  now  operdl*Ot;»i.  concepts  cannot 
be  accounted  for  by  applying  a  factor  tv  _  single-point 
cost  estimate.  The  estimate  of  the  decree  of  tech¬ 
nological  uncertainty  associated  with  a  given  weapon  is 
at  best  fairly  arbitrary,  and  to  present  only  a  single 
factor  rather  than  a  range  of  possible  values  is 
presumptuous. 
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Even  if  an  adequate  data  base  did  exist  for  the 
derivation  of  adjustment  factors  for  all  the  elements 
of  total  system  cost,  the  technique  would  still  be 
cumbersome,  time  consuming,  and  generally  unsuitable 
for  use  by  a  small  costing  group, 
b.  Cost  Sensitivity  Analysis 
(1)  Description 

Cost  sensitivity  analysis  is  the  systematic  examina¬ 
tion  of  the  changes  in  total  system  cost  of  a  weapon 
system  as  important  system  configuration  characteristics 
are  varied  over  their  relevant  ranges.  Configuration 
characteristics  include  hardware  characteristics,  system 
operational  concepts,  and  the  number  of  systems  to  be 
procured. 

In  systems  analysis  studies  wherein  alternative 
systems  and  different  configurations  of  the  systems  arc 
evaluated,  cost  sensitivity  analyses  are  helpful  ir. 
allowing  for  requirements  and  technological  uncertain¬ 
ties.  Co3t  sensitivity  analysis  can  be  used  to  establish 
th*  possible  range  of  cost  estimates  for  a  future 
weapon  system  whose  ultimate  configuration  characteris¬ 
tics  are  uncertain.  The  analysis  should  indicate  which 
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particular  configuration  characteristics  are  relatively 
less  sensitive  to  total  system  cost.  This  information 
is  especially  useful  when  total  system  cost  is  rela¬ 
tively  insensitive  to  the  most  uncertain  configuration 
characteristics. 

Figure  3  presents  sensitivities  and  insensitivities 
of  various  configurations  of  a  hypothetical  missile 
system.  In  the  figure,  the  total  system  cost  is  shown 
to  b.»  relatively  insensitive  to  increases  in  payload, 
possioly  because  many  elements  of  total  system  cost, 
such  an  ground- support  equipment  and  installations,  do 
not  chai  ge  significantly  as  missile  gross  weight 
increases.  However,  the  total  system  cost  is  sensitive 
to  the  type  of  propellant  used  in  the  missile.  Solid 
propellants  a.*e  less  expensive  to  store  and  handle  than 
cryogenic  propellants. 

Figure  4  illustrates  a  hypothetical  satellite 
system's  cost  sensitivity  to  the  probability  of  suc¬ 
cessful  launch  and  to  the  length  of  time  the  satellite 
will  remain  in  orbit.  If  the  objective  is  to  maintain 
a  fixed  number  of  satellites  in  orbit  over  a  given 
length  of  time,  total  system  costs  would  be  very 


(Fixed  Number  of  Missiles) 


Number  of  Satellites  in  Orbit) 


sensitive  to  the  launch  reliability  and  mean- time- to- 
failure  of  the  system.  The  sensitivity  of  various 
elements  of  the  total  system  cost  to  launch  reliability 
ai.d  mean- time-to- failure  also  could  be  shown  in  the 
same  way. 

(2)  Analysis 

Cost  sensitivity  analysis  provides  a  range  of 
system  costs  which  is  likely  to  be  a  more  realistic 
guide  than  a  single,  most  probable  cost.  However, 
there  are  limitations  to  consider.  This  technique 
requires  much  more  work  than  does  a  single-point  esti¬ 
mate.  Sometimes  it  ,s  difficult  to  present  a  concise 
and  clear  summary  of  all  the  numbers  generated  in  an 
extensive  sensitivity  analysis.  This  is  particularly 
true  when  the  consumer  of  the  cost  estimates  is  not 
accustomed  to  applying  the  results  of  such  an  analysis. 
This  technique  does  not  furnish  any  formal  measures  of 
uncertainty  as  a  basis  for  probability  statements.  Of 
course,  there  is  no  guauntee  that  any  given  cost 
analysis  will  include  all  the  relevant  ranges  of 
characteristics  or  possibilities. 


Although  cost  sensitivity  analysis  does  provide  a 
means  to  account  for  most  of  the  elements  of  require¬ 
ments  uncertainty,  it  does  rot  directly  state  the 
technological  uncertainties  of  the  weapon  system  nor 
the  possibility  of  a  change  in  enemy  capabilities.  On 
the  other  hand,  it  does  account  for  such  uncertainties 
indirectly  by  providing  costs  for  a  range  of  possible 
system  characteristics.  However,  for  any  one  of  the 
many  configurations  analyzed,  a  certain  technological 
advancement  is  decided  upon  as  most  likely  representing 
the  technical  requirements  for  the  system.  Then, 
assuning  that  the  operational  requirements  for  the 
system  do  not  change,  the  analysis  would  net  indicate 
the  increase  in  cost  if  unforeseen  technical  diffi¬ 
culties  were  encountered.  If  operational  char acteristics 
are  compromised  to  avoid  or  lessen  the  effect  of  the 
tecnrical  difficulties  encountered,  the  cost  sensitivity 
analysts  would  provide  cost  estimates  for  the  new  con¬ 
figuration.  Por  example,  some  unforeseen  technical 
problems  may  arise  in  achieving  the  accuracy  specified 
for  the  guidance  system  of  an  advanced  missile  system 
because  of  the  weight  Imitations  imposed  on  the  guidance 
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package.  More  money  may  be  spent  to  sclve  the  technical 
problems  that  have  preve:  '  ;  fulfillment  of  the  per¬ 

formance  requirements.  Cost  sensitivity  analysis  does 
net  predict  this  unexpected  expenditure.  However, 
should  the  weight  specified  for  the  guidance  package  be 
increased  to  avoid  large  expenditures,  cost  sensitivity 
analysis  would  provide  estimates  for  the  new  configura¬ 
tion.  Moreover,  if  seme  technical  breakthrough  were 
achieved,  cost  estimates  would  be  available  for  the  new 
ii'oroved  weapon  system. 

Certain  characteristics  of  cost  sensitivity  analysis 
should  be  recognized.  Usually,  more  time  is  needed  to 
prepare  a  wide  range  of  cost  estimates  than  a  single¬ 
point  estimate.  However,  the  likelihood  that  the  single¬ 
point  estimate  will  be  reliable  is  extre»aely  remote 
and.  therefore,  the  additional  tine  is  well  spent.  Cost 
sensitivity  analysis  does  not  require  any  long  or  com¬ 
plicated  cost-estimating  procedures  and  can  be  used  by 
a  small  group  of  cost  analysts  without  computers. 

In  conclusion,  cost  sensitivity  analysis  does 
provide  a  technique  which  accounts  tor  much  of  the 
re»_  -iremer.ts  uncertainty  by  presenting  a  range  of  cost 


estimates  for  sr any  contingencies.  It  is  considered  one 
c'f  the  rest  available  techniques  for  improving  cost 
sstirates  cr  future  weapon  systems, 
c.  Sufiective  Probability  distributions 
(1)  Pey-crict  icn 

Avctber  technique  for  expressing  uncertainty  in  cost 
estimates  allows  tie  cost  analyst  to  apply  his  feelings 
of  uncertainty  concerning  total  system  costs.  Thi  » 
technique  does  net  show  total  cost  uncertainty  related 
to  factors  the  analyst  has  neglected  to  consider  or  to 
changes  in  assumptions.  It  is  useful  for  expressing 
the  cost  uncertainty  remaining  after  the  cost  analyst 
Las  defined  the  system  and  identified  assumptions  ns 
carefully  as  possible.  The  objective  is  to  genei ate  a 
quantitative  statement  about  the  range  of  reasonably 
likely  values  for  total  system  cost.  It  is  not  recom¬ 
mended  that  a  statement  be  made  of  only  absolute  upper 
and  lower  limits  of  the  total  system  cost.  Instead, 

subjective  probability  concepts  should  be  used  to 
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describe  information  gathered  Lv  the  analyst  for  each 
eleaent  of  the  total  system  cost.!;  Oncer tainty  is 
quantified  by  defining  o  distribution*  m  ‘lecting  the 


relative  probable  values  of  all  elements.  Standard 
deviations  for  all  system  clenents  can  then  be  computed 
and  used  to  determine  the  standard  deviation  of  the 
torsi  system  cost . 

To  apply  this  t  ‘chniquc,  it  is  first  necessary  to 
establish  ground  rules  for  the  coEt.  estimate,  and  then 
to  use  the  best  data  and  cost-estimating  relations 


available.  Next,  the  estimate  is  reviewed  to  identify 
the  elements  of  the  total  cost  which  reflect  various 
technical  and  operational  uncertainties  Random 
variables  are  defined  to  represent  the  uncertain  cost 

elem(  ir.  The  subjective H  distribution  for  each  of 

.  *  •  • 

the  :  ndom  variables  is  defined  when  a  set  of  parameters 


for  any  of  several  distributions  is  specified.  By 


using  formulae  associated  with  each  representative  , 
distribution,  the  standard  deviation  and  mean  of  each 
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random  variable  is  computed  fran  the  parameter  values. 


Finally, /with  knowledge  of  how  system,  cost  elements 
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were  combined,  the  standard  deviation  ar*d  mean  of  thi 
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indicate  his  feelings  of  uncertainty  about  each  cost 
element.  Thex efox u ,  uuiinodal  distributions  arc  recom¬ 
mended  which  the  cost  analyst  can  readily  use  by 
specifying  only  two  or  three  parameters  for  each  random 
variable.  The  greater  the  number  of  parameters  that 
require  specification,  the  greater  the  flexibility  the 
analyst  has  in  defining  nis  feelings.  For  each  cost 
element,  the  analyst  s?iould  select  that  representative 
distribution  which  most  closely  resembles  his 
impressions  about  the  uncertainty. 

For  example,  if  th.i  cost  analyst  chose  a  normal 
distribution,  he  could  express  his  feelings  by  specify¬ 
ing  the  mode  and  the  points  which  bind  the  central, 
80-percent  portion  of  the  probability  distribution.  In 
a  symmetric  distribution,  the  mode  is  the  mean,  and 
both  points  must  have  the  same  deviation  from  the  mode. 
The  80-percent  range  is  arbitrarily  selected  as  con¬ 
venient  for  the  cost  analyst  to  attempt  to  specify. 

This  procedure  provides  an  idea  of  the  standard  devi¬ 
ation  of  the  cost  estimate.  Other  distributions  that 
might  be  used  are  log  normal,  three-  and  four-parameter, 
linearly  scaled  Beta  distributions,  and  trapezoidal 


distribution.  When  sufficient  quantitative  empirical 
data  are  available,  the  standard  deviation  and  mean  may 
be  computed  directly  without  the  use  of  representative 
distributions. 

A  simplified  example  of  this  technique  is  the  deter¬ 
mination  of  the  cost  of  placing  a  number  of  satellites 
in  orbit.  Two  independent  variables  are  the  cost  of  the 
booster  and  the  cost  of  the  satellite.  A  four-parameter, 
linearly  scaled  Beta  distribution  offers  a  wide  variety 
of  shapes  that  can  represent  uncertainty  in  cost  esti¬ 
mates  (Reference  7:  pp.  1-11).  The  extreme  limits, 
the  mode,  and  the  length  of  the  central,  80-percent 
range  are  specified  for  each  of  the  variables.  If  it 
is  80-percent  probable  that  the  actual  cost  will  lie 
between  $60,000  and  $100,000,  the  length  of  the  central 
range  is  $40,000.  The  values  expressed  below  are  in 
millions  of  dollars. 

Variable  Extreme  Limits  Mode  Central  Range 


Booster 

Satellite 


2.4  to  3.8 
0.05  to  0.5 


2.9 

0.2 


0.8 

0.1 


The  mean  (u)  and  the  standard  (a)  deviation  of  the  two 
cost  elements  are: 

U  o 

Booster  12.0  1.2 

Satellite  6.5  1.3 

Since  it  has  been  assumed  that  booster  cost  and  satel¬ 
lite  cost  are  not  interdependent,  the  mean  and  standard 
deviation  of  the  total  cost  are: 

liT  12.0  +  6.5  «  18.5 

oT  =  (1.2)®  +  (1.3)*  =  1.77 

If  the  total  cost  is  normally  distributed,  the 
probability  of  an  error  in  excess  of  $1,77  million  or 
one  o  is  1/3.  The  probability  of  an  error  in  excess  of 
$3.56  million  (or  about  19  percent  of  the  mean)  is  1/20 
These  values  are  the  probabilities  that  a  random  vari¬ 
able  is  more  than  Ko  away  from  the  mean 
(Reference  7:  p.  11). 

The  question  is  whether  or  not  it  is  worth  the 
effort  to  include  a  standard  deviation  for  the  total 
system  cost.  Additional  information  is  required  from 


the  analyst,  and  many  additional  steps  are  needed  for 
the  computation  of  the  total  system  cost.  The  average 
cost  analyst  may  be  reluctant  to  select  and  use  the 
distributions,  and  the  number  of  routine  calculations 
needed  makes  the  use  of  a  computet  almost  mandatory. 
Possible  distributions  and  their  characteristics  would 
have  to  be  readily  available  to  the  cost  analyst.  If 
uncertainty  is  not  described  statistically,  the  analyst 
requires  far  l^ss  information,  but  the  results  can  be 
easily  misused  if  uncertainty  is  not  quantified.  It  is 
more  realistic  to  quantify  the  uncertainty  for  each 
cost  element  than  to  do  the  same  for  total  system  cost, 
if  the  incremental  time  and  funds  required  are  available. 
(2)  Analysis 

Subjective  probability  distributions  are  especially 
suited  for  expressing  the  technological  uncertainty 
associated  with  the  advanced  weapon  system.  It  is  not 
clear  how  the  technique  could  be  used  to  express  the 
uncertainties  of  enemy  capabilities  and  changing 
operational  concepts. 

In  applying  this  technique,  the  cost  analyst  is 
called  upon  to  provide  his  estimates  of  the  four 
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required  parameters  and  to  select  a  distribution  that 
adequately  reflects  his  feelings  for  the  requirements 
uncertainty.  Many  additional  steps  are  needed  for  the 
computacion  of  the  tctal  system  cost.  Cost  analysts 
with  engineering  backgrounds  may  not  be  receptive  to  a 
technique  requiring  the  selection  and  use  of  probability 
distributions.  Unless  a  computer  is  used,  this  tech¬ 
nique  would  r. quire  too  much  time  for  a  small  group  of 
analysts.  The  number  of  routine  calculations  involved 
makes  the  use  of  a  computer  almost  mandatory  for  any 
size  costing  group, 
d •  Cost-Estimating  Relations 
(1)  Description 

Because  historical  cost  data  are  scarce,  standard 
statistical  techniques  can  seldom  be  used  in  estimating 
the  costs  of  future  weapon  systems.  However,  even 
limited  data  can  be  used  as  empirical  evidence  for  the 
derivation  of  cost-estimating  relations.  A  cost¬ 
estimating  relation  is  merely  a  statement  of  how  one  or 
more  variables  affect  another. 
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One  of  the  biggest  difficulties  in  gathering 
empirical  data  is  that  costs  are  recorded  by  various 
organizations  under  many  different  and  vaguely  defined 
cost  elements.  Available  must  be  arranged  in  such 

a  manner  that  cost-estimating  relations  can  be  derived. 
Establishing  a  data  base  is  a  continuing  job,  and  cost¬ 
estimating  relations  may  change  as  more  data  are  accu¬ 
mulated.  The  fundamentals  of  deriving  cost-estimating 
relations  are: 

-1  Identification  of  the  variables 

-2  Establishment  of  the  appropriate  functional  form 
-3  Recognition  of  the  constraints 
-4  Determination  of  the  confidence  placed  in  the 
estimating  relations. 

Correlation  and  regression  analyses  are  useful  in 
evaluating  these  relations. 

To  demonstrate  the  use  of  this  method,  the  costs 
will  be  estimated  for  development  engineering  and 
hardware  fc  brication  of  the  airframe  for  a  hypothetical 
advanced  booster  system.  The  THOR  booster  will  be  used 
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as  the  analog  for  the  estimation.  The  design  and 
development  cost  for  THOR  may  be  represented  as: 

C  =  4. OX 

where 

C  =  cost  in  millions  of  dollars 
4.0  =  THOR  engineering  time  in  millions  of 
man-hours 

X  =  cost  per  engineering  hour  in  dollars 

The  airframe  for  \-.he  hypothetical  booster  system  is 
assumed  to  be  different  from  the  THCR  booster  in  several 
respects.  The  new  system  is  larger,  has  different 
propellants,  and  requires  technological  advancements. 
Each  of  these  differerces  probably  means  an  increase 
in  the  engineering  hours  required. 

A  scaling  factor  can  be  derived  that  takes  into 
account  the  airframe  requirements  imposed  by  the 
chemical  composition  of  the  more  advanced  propellants. 

A  scaling  factor  can  also  be  used  to  adjust  for  the 
difference  in  weight.  The  ratio  between  the  propellant 
weight  of  the  new  vehicle  and  that  of  THOR  is  a  useful 
parameter,  but  cannot  be  used  directly  because  the  cost 
is  not  directly  proportional  to  the  weight  of  the 
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propellant  carr.ed.  Instead,  an  estimate  is  made  of 
the  percent  of  this  ratio  which  would  adequately  account 
for  differences  in  the  sizes  of  the  two  boosters.  Such 
a  relation  is  shown  below,  assuming  no  advance  in  the 
state-of-the-art : 

C  -  4. OX  [Y (0 . 9  +  0.1  £) ] 

b 

where 

C  *  cost  estimate  in  millions  of  dollars 
4.0  *  THOR  engineering  time  in  million*  of 
man-hours 

X  *  cost  per  engineering  hour  in  dollars 
Y  *  propellant  factor  (chemical  composition) 
a  **  propellant  weight  of  new  booster 
b  =  propellant  weight  of  THOR. 

In  the  above  relation,  advances  in  the  state-of-the- 
art  have  not  been  taken  into  account,  except  as  reflected 
in  the  propellant  factor  which  is  a  measure  of  the  com¬ 
plications  of  using  new  combinations  of  fuels  and 
oxidizers.  For  a  given  vehicle  design,  it  is  difficult 
to  adjust  for  an  advance  in  the  state-of-the-art  by 
manipulating  the  term  in  the  relation  which  accounts 
for  the  difference  in  propellant  weight.  For  example, 
the  expression  (0.9  +  0.1  a/b)  might  have  been  written 


(0.5  +0.5  a/b)  if  a  significant  advancement  in  tech¬ 
nology  were  required.  The  selection  cf  coefficients  in 
this  case  is  arbitrary  (Reference  8s  pp.  13-20). 

The  subjective  reasoning  that  goes  into  accounting 
for  advances  in  the  state-of-the-art  of  airframes  can 
be  quantified  by  plotting  a  family  of  curves  indicating 
propellant  weight  versus  the  ratios  of  airframe  weight 
to  various  propellant  weights.  The  curves  shown  in 
Figure  5  are  based  on  the  current  state-of-the-art  and 
on  the  assumption  that  a  relatively  minimum  weight  of 
tank,  structure,  and  miscellaneous  subsystem  equipment 
exists  for  any  given  propellant  weight.  A  proposed 
design  using  propellant  A  might  have  an  airframe 
weight  of  5,000  pounds  and  a  propellant  weight  of 
250,000  pounds,  producing  a  ratio  of  0.02.  From  the 
appropriate  curve  in  Figure  5,  it  can  be  seen  that  the 
standard  ratio  of  airframe  weight  to  propellant  weight 
ir.  0.03  for  propellant  A.  The  measure  of  complexity 
is  the  ratio  of  the  proposed  to  the  standard,  or 
0.03  to  0.02.  Thi'*  scaling  factor  of  1.5  could  be 
included  in  the  cost-estimating  relation  to  reflect,  at 
least  partially,  the  increase  in  engineering  man-hours 
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~ecessary  to  achieve  the  structural  weight  factor  for 
the  proposed  system.  The  lower  the  airfra  weight  for 
a  given  propellant  weight.,  the  more  difficult  the 
development . 

The  greatest  disadvantage?  of  techniques  utilizing 
cost-estimating  relations  result  from  a  general  lack  of 
an  adequate  data  base  from  past,  present,  and  projected 
programs.  Some  reasonably  applicable  data  do  exist 
for  several  aircraft  programs,  but  little  are  available 
on  missiles,  and  almost  no  data  are  available  on  satel¬ 
lite  systems.  In  some  crses,  a  simple  factor  relation 
may  exist  that  can  be  ex  ressed  as  a  single  number. 

For  example,  a  factor  for  pay  and  allowances  of  per¬ 
sonnel  is  readily  avail  ble  and  easy  to  apply.  On  the 
other  hand,  estimating  relations  can  be  much  more  com¬ 
plicated  and  difficult  to  develop  and  apply  quickly. 

(2)  .Analysis 

Although  certain  terms  in  the  cost-estimating  rela¬ 
tions  may  allow  the  cost  analyst  to  apply  scaling 
factors  which  could  account  for  technological 
advancements  required  by  the  system,  the  cost-estimating 
relations  in  themselves  do  not  state  requirements 
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uncertainty.  They  can  indirectly  reflect  uncertainty 
as  the  cost  analyst  interprets  the  available  data 
before  using  the  estimating  relations.  Some  cost¬ 
estimating  relations  are  simple  to  develop  and  apply; 
others  are  complex  and  involve  lengthy  calculations. 

A  small  group  of  cost  analysts  could  easily  apply  many 
of  the  simple,  well-established  estimating  relations, 
but  it  probably  would  not  have  the  time  nor  the  data  to 
derive  any  new  relations.  Since  the  concern  herein  is 
Air  Force  systems  planned  for  8  to  10  years  in  the 
future,  cost-estimating  relations  already  in  existence 
would  probably  not  be  useful. 

D*  The  l."o?.t  Satisfactory  Technique — Cost  Sensitivity 
Analysis 

The  technique  most  likely  to  improve  cost  estimates  of 
future  weapon  systems  is  cost  sensitivity  analysis.  Of  the 
four  techniques  evaluated,  cost  sensitivity  analysis  best 
provides  for  requirements  uncertainty.  It  can  be  used 
effectively  by  a  small  group  of  cost  analysts  and  does  not 
require  extensive  computer  facilities. 

One  proponent  points  out  that  although  cost  sensitivity 
analysis  was  not  developed  initially  to  help  cope  with  the 
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uncertainty  problem  per  se,  it  can  be  useful  in  this  regard 
(Reference  4:  pp.  23-28).  For  example,  it  may  aid  in 
determining  cost-estimating  uncertainty.  For  this  purpose, 
the  requirements  uncertainty  would  be  assumed  to  be  zero; 
then,  the  sensitivity  analysis  could  be  used  to  show  how 
system  cost  varies  because  of  uncertainties  in  cost- 
estimating  relations,  errors  in  basic  data,  extrapolation 
errors,  and  the  like.  This  method  could  be  used  to  solve 
an  uncertainty  about  the  price  of  a  type  of  rocket  propellant. 
The  costs  of  future  boosters  which  require  the  propellant 
could  be  estimated  on  the  basis  of  several  propellant  prices, 
rather  than  on  one.  As  a  result,  the  sensitivity  of  total 
equipment  cost  to  the  probable  range  of  prop-allant  prices 
would  be  indicated. 

No  technique  is  perfect,  and  cost  sensitivity  analysis 
is  no  exception.  It  does  not  furnish  any  formal  measures 
of  uncertainty  and  thus  does  not  provide  the  basis  for 
probability  statements.  The  many  numbers  generated  in  an 
extensive  cost  sensitivity  analysis  must  be  carefully  pre¬ 
sented  to  the  group  synthesizing  the  costs  with  the  measures 
of  effectiveness.  Of  even  greater  importance  is  the 
presentation  of  the  cost-effectiveness  analysis  to  the 
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Air  Force  decision  maker.  Here  it  is  especially  signifi- 
cant  to  indicate  all  of  the  possible  contingencies 
associated  with  the  cost  analysis  so  that  better  decisions 
can  be  made. 

Cost  sensitivity  analysis  requires  much  more  work  than 
a  single-point  estimate,  and  there  is  no  guarantee  that  all 
relevant  possibilities  will  be  included.  Regardless  of  its 
limitations,  cost  sensitivity  analysis  is  probably  one  of 
the  best  available  techniques  for  dealing  with  the  uncer¬ 
tainty  problem  in  cost  analysis  of  future  systems  and 
forces.  It  fits  well  with  the  over-all  objectives  of  cost 
analysis,  since  the  sensitivity  approach  is  the  one  that  is 
most  useful  to  weapon-system  analysis  studies  and  other 
activities  requiring  cost  inputs. 

The  cost  sensitivity  analysis  approach  applies  equally 
well  in  a  great  many  circumstances  and  at  many  decision 
levels.  It  is  by  no  means  limited  to  the  defense  field. 

In  the  automotive  industry,  cost  sensitivity  analyses  are 
performed  to  guide  the  selection  of  designs  for  future 
transmissions  and  other  automobile  parts.  This  type  of 
study  is  worthwhile  in  the  automobile  industry  because  ov 
the  high  volume  of  the  product  which  will  be  manufactured. 
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Cost  sensitivity  analysis  is  applied  in  electronics  system 
design,  where  the  implications  of  reliability,  durability, 
or  accuracy  levels  for  a  computer  or  radar  set  can  be 
measured.  It  can  be  very  useful  in  determining  the  cost 
implications  of  alternative  designs  for  a  particular  com- 
ponent;  for  many  people,  this  type  of  analysis  is  more 
important  than  the  over-all  sensitivity  of  the  total  costs 
to  these  design  alternatives. 

Cost  sensitivity  analysis  is  an  attempt  to  estimate  the 
financial  magnitude  of  alternative  weapon-system  mixes, 
designs,  and  operational  concepts,  and  the  time  phasing  of 
these  alternatives.  Cost  sensitivity  analysis  is  not  a 
complete  study  in  and  of  itself,  but  is  part  of  a  full 
operations  research  or  systems  analysis  effort.  If  it  is  to 
be  useful,  it  must  be  integrated  into  the  over-all  cost- 
effectiveness  evaluation. 

Some  critics  of  cost  sensitivity  analysis  point  out  that 
the  cost  analyst  specifies  a  likely  range  for  each  of  the 
cost  elements  and  then  totals  the  lowest  values  and  the 
highest  values  for  all  the  coat,  elements  to  indicate  a  range 
of  total  system  costs.  In  fact,  the  possibility  is  very 
remote  that  all  elements  will  actually  attain  their  lowest 
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or  their  highest  values.  Hence,  such  a  range  of  values  is 
an  overstatement  of  the  magnitude  of  likelv  variability  in 
total  system  costs.  However,  this  criticism  applies  to 
cost  sensitivity  analysis  as  it  is  used  to  cope  with  cost¬ 
estimating  uncertainty,  and  does  not  detract  from  the 
usefulness  of  cost  sensitivity  in  accounting  for  requirements 
uncertainty. 

Another  expert  in  the  field  of  cost  analysis  of  future 
weapon  systems  states  that  cost  sensitivity  analysis  is  a 
useful  technique  for  dealing  with  problems  of  uncertainty, 
the  more  so  because  conventional  statistical  methods  for 
deriving  confidence  limits  and  other  measures  of  uncertainty 
cannot  be  applied  to  cost  estimates  (Reference  4:  p.  20). 
First  of  all,  to  derive  the  conventional  statistical 
measures  of  uncertainty,  a  representative  sample  must  be 
drawn  from  a  designated  population  and  used  as  a  basis  for 
the  statistical  analysis.  In  the  cost  analysis  of  future 
systems,  a  large  population  is  not  available.  On  the  rare 
occasions  when  samples  can  be  drawn,  the  size  of  the  sample 
is  invariably  very  small,  making  the  application  of  most 
statistical  theory  impossible.  In  studying  proposed  future 
systems,  numerous  uncertainties  must  be  recognized  together 
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with  their  impact  on  system  costs.  Studies  of  historical 
cost  data  have  shown  that  perhaps  the  most  important  reason 
for  differences  between  early  estimates  and  final  costs  is 
that  the  ultimate  configuration  of  the  system  was  considera¬ 
bly  different  from  that  envisaged  early  in  the  program. 

Cost  sensitiviti  analysis  deals  explicitly  with  cost 
differences  related  to  differences  in  system  configuration, 
and  therefore  can  provide  a  range  of  system  costs,  which  is 
likely  to  be  a  more  realistic  guide  than  a  single,  most 
probably  cost. 

If  extensive  cost  sensitivity  analyses  are  required, 
the  capacity  of  a  small  group  of  analysts  may  be  exceeded. 
Therefore,  it  behooves  the  cost  analyst  and  the  Air  Force 
to  avoid  broadening  the  scope  of  the  analysis  beyond  that 
which  is  essential  for  making  sound  decisions. 

However,  there  may  be  occasions  when  the  importance  and 
scope  of  the  study  necessitate  the  renting  of  computer  time. 
Under  these  conditions,  it  would  bo  important  to  reach  an 
agreement  with  the  consumer  concerning  the  cost  of  the  com¬ 
puter  time  as  compared  to  the  value  of  the  analysis. 

Possibly  the  Air  Force  could  arrange  for  free  computer  time, 
or  could  include  such  expenditures  in  the  contractor's  fee. 
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£.  Application  of  Cost  Sensitivity  Analy.si-B 

The  cost  of  a  hypothetical  advanced  weapon  system  is 
analyzed  in  the  following  paragraphs  to  damonstrate  the 
application  and  limitations  of  cost  sensitivity  analysis. 

The  Directorate  of  Development  Plena,  Headquarters 
United  States  Air  Force,  sends  a  formal  request  to  a  non¬ 
profit  corporation  for  a  cost-effectiveness  study  of  a 
family  of  advanced  ICBM's.  The  purpose  of  the  study  is  to 
examine  a  wide  range  of  payload  weights  that  can  be  delivered 
by  the  missiles  to  determine  which  size  of  payload  is  the 
most  cost  effective. 

1 .  Assumptions 

The  following  assumptions  are  made  to  simplify  the 
analysis: 

-1  The  booster  system  and  corresponding  propellants  to 
be  used  are'well  within  the  present  state-of-the-art, 
and  the  same  type  of  booster  is  used  regardless  of 
the  size  of  the  payload. 

-2  The  same  general  type  of  re-entry  vehicle  is  used 
for  all  payloads. 

-3  Two  basing  postures  are  considered-— hard- fixed  and 
truck-mobile. 
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-4  The  guidance  system  is  an  advanced  design  and  is 
well  beyond  current  technology. 

-5  A  fixed  number  {100)  of  missiles  is  to  be  procured. 
As  a  result  of  these  assumptions,  payload  weight  and  the 
basing  posture  are  variables,  and  all  other  elements  of 
weapon-syscem  design,  development,  and  operation  remain 
constant . 

2 .  Costs  of  Pavload  Weight  and  Basino  Posture 

Table  7  is  a  sample  format  for  displaying  costs  of  many 
elements  of  the  total  system.  Four  payload  weights  are 
considered.  System  costs  are  prepared  for  the  two  basing 
postures  for  each  payload  weight.  The  major  cost  categories 
considered  are  research  and  development,  initial  investment, 
and  5  years  of  operation.  The  5  years  of  operation  is 
arbitrary,  since  weapon-system  life  can  vary  from  1  to 
10  years.  Usually,  several  system  lifetimes  are  assumed, 
and  the  results  are  shewn  for  the  various  assumptions. 

A  hypothetical  set  of  results  of  the  sample  cost 
analysis  is  shown  in  Figure  6.  Payload  wevght  versus  total 
system,  cost  is  plotted  for  the  hard- fixed  and  truck-mobile 
basing  postures.  Probable  cost  limits  are  also  indicated 
for  the  nard- fixed  posture.  The  results  indicate  that 
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SENSITIVITY  OP  COST  TO  PAYLOAD  WEIGHT  AND  BASING  POSTURE 


total  system  cost  is  fairly  insensitive  to  the  range  of 
payload  weights  considered  but  is  very  sensitive  to  the 
type  of  basing  employed  by  the  missile.  The  development, 
procurement,  and  operation  for  100  of  the  hard- fixed 
missiles  for  5  years  would  cost  a  total  of  about  $2  billion. 
The  truck-mobi  le  system,  with  the  same  size  payload,  would, 
cost  about  twice  as  much  as  the  hard- fixed  missile. 

The  kinds  of  information  presented  in  a  completed 
Table  7  and  in  Figure  6  are  especially  useful  to  the  Air 
Force  decision  maker  who  is  attempting  to  select  the  best 
weapon  system  from  among  several  possibilities.  For  example, 
the  decision  maker  who  has  a  fixed  number  of  dollars  to 
procure  the  best  weapon  sy  tern  car-  see  immediately  from  the 
cost  results  that  he  can  obtain  roughly  twice  as  many  hard- 
fixed  ICBM's  as  truck-mobile  ICBM's.  Then  he  can  consider 
incremental  advantages  such  as  the  increased  survivability 
obtainable  frcm  a  mobile  system.  Cost  estimates  developed 
from  cost  sensitivity  analyses  are  readily  adaptable  to  a 
variety  of  situations  and  can  be  meshed  with  the  various 
measures  of  effectiveness.  N 

Cost  information,  as  presented  in  Figure  6,  permits  the 
user  to  assess  quickly  the  cost  implications  of  m&r*y  of  the 
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causes  of  requirements  uncertainty.  For  example,  if  recent 
intelligence  information  indicated  that  the  potential  enemy 
had  developed  a  means  of  destroying  the  hard-fixed  ICBM's, 
the  decision  maker  would  have  available  cost  data  on  an 
alternative  basing  posture,  as  shown.  Improved  ratios  of 
warhead  yield-to-weight  might  be  achieved  through  under¬ 
ground  nuclear  testing.  If  warheads  could  be  developed 
with  higher  yield  for  the  same  weight,  a  smaller  missile 
payload  weight  would  provide  the  same  destructive  potential 
Again,  Figure  6  indicates  the  possible  cost  saving  of 
selecting  a  smaller  payload  ICBM. 

3-  Costs  of  Boosters  and  Propellants 

The  assumption  was  made  that  existing  booster  and  pro¬ 
pellant  technology  would  be  used  for  the  missile  system. 

Fairly  accurate  cost  data  should  be  available  for. items 
like  those  which  hav 3  been  developed  and  tested.  Cost¬ 
estimating  relations  are  probably  available  for  determining 
the  booster  and  propellant  costs  as  a  function  of  the  weights 
required  for  the  delivery  of  the  payloads  under  consideration. 
4.  Costa  of  Guidance  System 

The  guidance  system  to  be  used  in  the  family  of  ICBM's 
is  assumed  to  be  an  advanced  design  and  is  well  beyond  the 
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current  technology.  In  this  case,  no  historical  data  are 
available  and  cost- estimating  relations  have  not  been 
derived.  The  cost  analyst  is  faced  with  the  dilemma  of 
costing  a  highly  complex  system  that  is  described  only  in 
general  terms  and  may  even  be  impossible  to  develop.  In 
Table  7,  the  cost  of  the  guidance  system  is  hidden  in 
several  of  the  subcategories  of  total  system  cost.  For 
example,  it  accounts  for  some  portion  of  the  research, 
development,  test,  and  evaluation  costs,  and,  of  course, 
the  procurement  costs  of  the  primary  equipment  which,  in 
this  case,  is  the  missile. 

What  can  the  cost  analyst  do  to  reflect  his  uncertainty 
about  the  development  and  procurement  costs  of  the  guidance 
system?  One  simple  solution  is  to  make  a  range  of  estimate 
based  upon  rhe  best  available  information.  The  highest, 
lowest,  and  most  likely  costs  of  developing  and  procuring 
the  advanced  guidance  system  could  be  estimated.  The  next 
question  to  be  answerr!  ic  what  effect  does  this  range  of 
estimates  hav~  on  the  ♦•*tal  system  cost?  If  the  highest 
estimates  for  the  gu  .Jince  system  have  a  negligible  effect 
on  total  system  cost.  t  probably  would  not  be  "ecessary  to 
indicate  the  ?ssociated  small  increases  in  cost.  However, 


if  total  system  cost  is  sensitive  to  the  cost  of  the 
guidance  syr  he  results  should  be  presented  to  the 

decision  ma;  . 

It  is  diriicult,  if  not  impossible,  for  the  cost  analyst 
to  prepare  estimates  covering  such  unexpected  events  as  a 
guidance  system's  failing  to  meet  system  requirements  or 
proving  to  be  impossible  to  develop  at  any  cost.  One  tech¬ 
nique  which  accounts  for  such  unr...  >  ;  ilnties  is  to  estimate 
the  rate  of  spending  required  to  develop  the  system's  com¬ 
ponents.  This  would  indicate  to  the  decision  maker  the 
amount  of  dollars  already  cx""r.;itted  at  the  time  when  the 
feasibility  of  developing  the  advanced  guidance  system 
should  be  verified.  Other  useful  information  would  be  the 
cost  rf  changing  to  an  existing  guidance  system,  should  the 
advanced  design  prove  impossible  to  develop.  Figure  7 
shows  a  typical  time  phasing  of  system  costs  for  the  major 
categories  of  research  and  development,  investment,  and 
operation.  The  point  marked  on  the  research  and  development 
curve  indicates  the  anticipated  date  for  determining  the 
feasibility  of  developing  the  advanced  guidance  system. 
Should  the  system  prove  technically  feasible,  the  solid 
lines  indicate  the  most  likely  system  costs  for  the 
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remainder  of  the  missile  program.  Should  the  guidance 
system  prove  to  be  impossible  to  develop,  then  the  missile 
program  might  be  revised  to  include  a  different  guidance 
package.  The  dashed  lines  in  Figure  7  show  the  effect  on 
the  system  costs  of  switching  to  a  different  guidance  system. 
Obviously,  the  money  already  spent  on  the  advanced  guidance 
system  would  be  lost,  and  the  effectiveness  of  the  missile 
system  probably  reduct'd. 

Unfortunately,  past  experience  with  weapon  programs 
demonstrates  that  changes  and  delays  in  weapon" system 
development  programs  cannot  be  contended  with  this  easily. 

For  example,  the  estimated  date  for  proving  the  feasibility 
of  the  new  guidance  system  could  slip.  This  might  cause  a 
series  of  costly  delays  in  the  rest  of  the  program.  After 
the  guidance  system  proves  feasible  to  develop,  it  may  still 
fail  to  meet  the  operatijnal  requirements.  This  may  lead 
to  more  costly  delays  and  expensive  modifications  of  the 
missile  system.  The  cost  analyst  cannot  anticipate  such  a 
wide  range  of  possible  events.  However,  by  presenting  good 
coot  sensitivity  analyses  and  by  carefully  outlining  the 
assumptions  behind  the  analyses,  he  can  help  the  decision 
maker  make  more  logical  selections  of  weapon  systems. 
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F.  Importance  of  Other  Techniques 


The  other  three  techniques  discussed  should  not  be  dis¬ 
regarded.  Adjustment  factors,  subjective  probability 
distributions,  and  cost-estimating  relations  are  all  useful 
techniques  in  special  situations.  During  the  cost  sensi¬ 
tivity  analysis  of  a  weapon  system,  one  or  more  of  the 

i 

other  techniques  are  almost  certain  to  be  helpful  in 
estimating  all  the  individual  elements  of  total  system  rost. 

There  may  be  occasions  when  combinations  of  cost- 
estimating  techniques  might  provide  better  estimates  than 
cost  sensitivity  analysis  alone.  For  example,  the  costing 
group  may  be  asked  to  respond  to  a  request  from  the 
Directorate  of  Development  Plano  in  a  matter  of  hours.  On 
this  occasion  there  would  not  be  sufficient  time  to  perform 
a  cost  sensitivity  analysis.  Instead,  the  costing  group 
would  probably  supply  the  best  single-point  estimates  that 
could  be  prepared  in  the  allotted  time  Probably  a  few 
contractor  reports  describing  the  system  would  be  the  only 
data  available  immediately.  Perhaps  one  or  two  contractor 
reports  might  contain  cost  data.  Usually,  only  the  major 
categories  of  total  system  cost  are  shown  in  contractor 
reports,  and  it  is  difficult  to  determine  which  subcategories 
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are  included  in  the  costs.  Many  times  the  assumptions  on 
which  the  cost  estimates  are  based  are  incomplete  or  omitted. 
Since  time  would  be  insufficient  to  perform  a  cost  sensi¬ 
tivity  analysis,  the  cost  analyst  would  have  to  apply  some 
other  technique.  In  this  circumstance,  the  cost  analyst 
should  review  the  contractor  costs  for  any  obvious  errors 
or  omissions.  Unless  otherwise  specified  by  the  Directorate, 
the  attempt  should  always  be  made  to  present  total  system 
costs.  Next,  the  analyst  should  review  past  cost  sensi- 
tivity  analyses  to  find  useful  data  on  similar  systems. 
Finally,  existing  cost-estimating  relations  ard  adjustment 
factors  could  bo  used  to  provide  the  quick  single-point  cost 
estimates.  The  limitations  and  assumptions  that  are  an 
integral  part  of  such  rough  estimates  must  be  carefully 
explained  to  the  user. 

Cost  sensitivity  analysis  does  not  provide  any  formal 
measures  of  uncertainty;  therefore,  it  does  not  furnish  the 
basis  for  making  probability  statements.  If  the  Air  Force 
requests  this  type  of  analysis,  some  other  technique  must 
be  utilized.  The  PERT  (Program  Evaluation  and  Review 
Technique)  is  used  to  cope  with  uncertainty  in  schedule 
estimates.  The  PERT  procedure  assumes  that  the 
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distribution's  standard  deviation  can  always  be  approximated 
satisfactorily  as  one-sixth  of  the  total  range  between  the 
highest  and  the  lowest  estimates.  However,  standard  devi¬ 
ations  calculated  in  this  manner  may  ba  unreliable,  and  no 
allowance  is  made  for  widely  dispersed  or  sharply  peaked 
distributions.  Hence,  the  cost  analyst's  actual  feelings 
might  be  distorted.  To  avoid  some  of  these  shortcomings, 
the  subjective  probability  distribution  technique  described 
earlier  is  recommended.  It  should  be  kept  in  mind  that  the 
successful  application  of  the  subjective  probability  dis¬ 
tribution  technique  depends  upon  a  good  background  in 
statistics.  In  addition,  automatic  data  processing  is 
desirable. 

There  are  many  occasions  when  combinations  of  cost¬ 
estimating  techniques  offer  better  solutions  than  could  be 
achieved  with  only  one  technique.  Por  example,  the  cost  of 
personnel  pay  and  allowances  is  included  in  annual  operating 
costs.  Cost-estimating  relations  are  well  established  for 
annual  pay  and  support  of  the  various  grades  of  military 
personnel  and  serve  as  useful  complements  to  the  cost 


sensitivity  analysis.  Other  elements  of  total  system  cost 
could  prove  to  be  difficult  to  handle  by  cost  sensitivity 
analysis.  In  such  cases,  adjustment  factors  and  subjective 
probability  distributions  could  be  of  help.  However,  cost 
sensitivity  analysis  is  the  single  most  effective  technique 
for  accurately  estimating  the  cost  of  future  weapon  systems. 


82 


I 


•w  rWfl’M  .  i  ' -"r— -r* >%. r  s  r  i  <»,  n  .ir.J" jr-m  ,rm  r.  j 


BIBLIOGRAPHY 


1.  AFR  80-1,  Research  and  Development,  Definition  of  Terms 

(Department  of  the  Air  Force,  Washington,  August  1959} . 

2.  Merton  J.  Peck  and  F.  M.  Scherer.  The  Weapons  Acqui¬ 

sition  Process:  An  Economic  Analysis  (Boston:  Harvard 
University,  1962). 


3.  A.  W.  Marshall  and  W.  H.  Meckling.  "Predictability  of 

the  Costs.  Time,  and  Success  of  Development,"  The 
Rate  and  Direction  of  Inventive  Activity:  Economic 
and  Social  Factors,  a  report  of  the  National  Bureau  of 
Economic  Research,  New  York  (Princeton,  New  Jersey: 
Princeton  University  Press,  1962,  pp.  467-469). 

•  $<•  ■>.  •  '• . 

4.  G.  H.  Pisher.  A  Discussion  of  Uncertainty  in  Cost 

Analysis  (RM-3071,  The  RAND  Corporation,^  April  1962) . 

.  ; 

5.  Charles  J.  Hitch  and  Roland  N.  McKean.  'The  Economics 

of  Defense  in  the  Nuclear  Ace  (R- 346,  The  RAND 
Corporation,  March  1960). 

* 


-  .  •  -V '  *'  Vz&Si# '' -c*  ' 

7.  S.  Sobel.  A_Computerized  Technique  to  Express  Uncer¬ 

tainty  in  Advanced  System  Cost  Estimates  (ESD-TR-65-79, 
The  MITRE  Corporation,  September  1963)  .  * 

,  V 

.  .  ’  - .  '*■  \  .  '  V*  *  '  * 

8.  J.  W.  Noah.  Identifying  and  Estimating  R&D  Costs 

(RM-3067-PR,  supplement,  The  RAND  Corporation, 

May  1962)  .  >.  V-  .  :  .\,  ■  '  ; 

-  >  . .  • 5 v 

9.  A.  Alchian.  Reliability  of  Cost  Estimates-^-Gowe 

Evidence  (RM-481,  The  RAND  Corporation’,  October  1950) . 


N  *  i*  .  i 


<  •  J 


: ;  '■  ■■  t ;;  ,  * 

'  '■;T  -J-'  '  :»  J  'V  vS  ‘  ' 

•  '.:V-  ■' 


iVil 


Wy 


83 


